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APPARATUS FOR MANIPULATING AN OBJECT 
DISPLAYED ON A DISPLAV DEVICE BY USING A TOUCH SCREEN 

■VM I Qi / . .r^r-pnimn O f Tl" tmvfhtion 

i7i»i Y"<' th " Tnventlon 

The present invention relates to an apparatus for 
manipulating an object displayed on a display device by UB.n, a 

"""VsTsrof computer systems for data processing has become 
widespread in recent years, more and more users are being 
10 required to handle same. i.e. , to input data and converse wxth 

dXa processors such as «orX stations and personal computers A 
vast variety of application programs are available for recent 
1? data processors and even a complicated application can be 
processed by using such application programs in combination 
« However, it is a great problem that such data processors are very 
" }i to handle, especially to manipulate an object disp ayed 

i;; on a display device, for those who have less than a substantial 

knowledge of computers. 
t3 Therefore, an apparatus for manipulating an object displayed 

' on a display device, which is easily handled even ^« - P-"" 
who has no special Knowledge of computers, is in great demand. 

n„=>-..<r»lr<on -f "I' ■'-l^ted Art 

rig. 1 illustrates a computer system with a conventional 

user interface. 

A computer system with a conventional user interface 
consists mainly of a central processing unit (CPU, ^ 
memory 5, a Keyboard/mouse 2, a frame memory 6 and a hard disK 
LterLcl 71, Which are interconnected via a ^''^^^J^l; ^^^^^^^ 
a hard disk 7 and a display unit 3, which are connected to system 
bus via the hard disk interface and the frame memory 6, 
resplctively. The main memory 5 stores a system control program 
and application programs which handle graphics P™-^^^^^ 
provides a work area for use by the programs. The CPU pert 
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a^ve syste.. an operator is retired to input ^J^'^'J'^^^^^^ 

manipulating the o.^ect by using a ^^^^^'"^^ ^jj^,^, 

an icon (a symbolic representation of a computer function) , 

an icon (a y , ^y using the keyboard/mouse 2, in 

displayed on a display unit 3 by ^ troublesome 

order to command a desired function. However, it 

Ind annoying to use a Xeyboard/mouse and icons and a person with 

l^L Tan a substantial Knowledge of computers tends to be 

v.«i„rtant even to touch a keyboard/mouse. 

te^erore, it is a great problem that such data processors 

are very difficult to handle for those who have less than 

substantial knowledge of computers. 

gillffl ^pv OF TO TMVEWTIOll 
It is an Object of the present invention to provide an 
apparatus Which can easily manipulate an object displayed on a 

'"'"xl If Ither Object of the present invention to provide a 
user interface with which a user can easily manipulate an object 
disDlayed on a display unit. 

TO achieve the above and other objects. /-^"^ 
invention provides a touch screen, a plurality of data files, 
Tspla inC t,„n storage means and display control means 

ll the apparatus for manipulating an object displayed on a 
aispla^ Tevic:: the touch screen, which is a "ansparent pane 

raro:rra::r:r puts touch ^creen informat. res-i^ - 

motion of the body. The plurality of ^^^"/^^^^ f I 

aata for displaying the ^^^J" -"Hl^t ; ^0^:^: al 
information storage means stores object inrorm 



device, file information which spec.fxes 
5 a part of the object data stored in one of ''"^/^""^^^^ 

files and a file name which specifies one of said P^""^^^^ ° 
laia nies. The display control .eans recognizes a .anxpulatxon 
to be conducted on the object, based on the touch screen 
InforB^tion fron the touch screen and on the object .nfor^at on 
. . ■ »h= disDlav Information storage means, and displays 
" Te Tb'-^nhrnay device in accordance with the aforesaid 
recognition. 

a p.T^ p nF.seR ^I"TnH nr THE nKftWiygS 

f; Fig. 1 illustrates a computer system with a conventional 

'^^^ Cnra blocK diagram of the configuration of a touch 
i; screen-!guipped workstation, in which the present invention is 

h n^I'ls a schematic diagram illustrating the principle of 

20'? the present invention; 

(5 Fig. 4(a) is a display information table; 

;j Fi,. 4(b) illustrates the format of touch-screen 

' '"'Tir=''iB a flowchart illustrating a piC manipulation; 

F igs. e(a) and e(b) are related and Interconnected diagrams 
illustrating and Fig. 6(c) Is a table explaining a picX 

manipulation; ^ table 

Fig. 7(a) is a diagram illustrating and Fig. 7(b) 

3„ -"''^ :,rand rr lilted diagrams illustrating and 
pia 8 fc) is a table explaining a push manipulation; 

" rlgl. 9(a) and 9(b) are related diagrams illustrating and 
Fig. 9(c) is a table explaining a flip manipulation; 
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Fig. 10(a) is a diagram illustrating and Fig. 10(b) 
table explaining a roll manipulation; ,v.n ^« « 

Fig. 11(a) is a diagram illustrating and Fig. 11(b) is a 
table explaining a distort-restore manipulation. 

Throughout the above-mentioned drawings, identical reference 
numerals are used to designate the same or similar component 
parts . 

pp.o^TT.a.TnM nv THE rrFFF^°^" imBODTMENTS 
Fig. 2 is a block diagram of the configuration of a touch 
screen-equipped workstation, in which the present invention is 

implemented. . t?4^ i i-h^ 

in addition to the conventional system shown in Fig. 1, the 
system of Pig. 2, for implementing the present invention, 
comprises an input-output (abbreviated to I/O) port 8a touch 
screen controller 15 and a touch screen unit 1 with a touch 
screen 11. The touch screen controller 15, connected to the 
input-output port 8 through an BS-232C interface, controls the 
touch screen unit 1. The touch screen unit 1, which is 
sensitive, at a position (defined by X-Y coordinates) at which 
Le touch screen 11 is touched and particularly to the pressure 
W applied thereon when so touched, acts as a user ^ 
y a!!ows a user to send signals to a computer by touching that area 
" thereon with an element, such as the finger, ... etc. 

Fig. 3 is a schematic diagram illustrating the principle of 

25 the present invention. ^ 4. ,4. 

For easy understanding of the principle, the input-output 
port 8, touch screen controller 15 and touch screen unit 1 shown 
in Fig. 2 are represented in Fig. 3, in the composite, by the 
touch screen unit 1 and the frame memory 6 and display unit 3 

30 and, as before-noted, the touch screen 11 actually is a 

transparent panel which is mounted on the display surface, or 
scr«n. Of L display unit 3 but is shown separately therefrom 
" Fig 3 Of Fig. 2 are represented in Fig. 3, in the composite, 
b^ t e'd splay unit 3. The hard disK interface 71 and hard dis. 
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.. J i» fit. 3 in the composite, by the 
7 of rig. 2 are represented in Tig- 3, in x. . . 

I H IL 7 A system controller 50, a touch discrxmxnator 51, a 
hard disk 7. A sy ^^^^ ^^^^^ ^^^^^ 5 

display 1 ^/^ constructed hy hardware, , 

ITs l , inro^rtion Lie it, stored in the main memory 5. 

clrot the display operations performed hy the present 

"'^^"Kt'4(a, Shows a display information tahle. Fi,. 4,h, 
snows touch screen i"'"^"""; provided in the 

The display ^^"^ i„,l„,es object type 

main memory 5 and corresponds objects i 
information, display position information file 
normal-display file name and special-state '^^^ 
type defines the type, including the shape, P^°f«'^^^' 
rrcumstances, etc., of the ob.ec. J^J^^^^^jTZ height 

ir:rth:r.:r ^to;^^^^^^^^^^^^ a. the^ng. 
a. !,;ich the ^:;:^zirr:z:^^^ 

reTan; :r r^s roUing to vi.- -.ct defines 

• t^AAth W height H) of the whole object relative to 

T.zv:Tsi::z position 

X V) of the object portion being displayed on the display 
dlviie, relative to the whole object whose data is stored in the 

/^•=„i,v file The normal-display file name specifies a 
normal ^-P^^^/^^^;^ displaying a normal state 

display l-^^''-';^^ special-state file name specifies a 

,1 a turn-over indication of display color, used for 

"e;;iri displaying an intermediate ; ^" 

manipulating the -^^^ ^ ^ Taril^PulI^ton^ 

.nunit 1 includes a touch position (X-Y coordinates) where 
Te toucrscrU 11 i. touched and the pressure applied thereon. 



r^^nr c wr .ne op«ato.s n„,er .as on ..e touC screen 
irthat is, a touch type including, e.g., a "continuous touch 
stlr^ ana "continuous touch end- as e:<plained later. The touch 
firc'lMinator 51 sends, to the system controller 50 the result 
o the touch discrimination performed tolXeen 
3K, Which includes touch type and, accordxn, to the touch 

«^ «!ets of touch coordinates. 
"-^l^^^Z:^ « .ro„ the touch discriminator 51 
and the display information tahle IT, the system --"^^^ =° 
determines the type of manipulation which was -"/"f^^ ^ 
operator and, according to the determination, updates "lisplay 
operator , controller 50 sends, to 

ZTX - o - ---^ --^^ " 

^^Isplay update data- which reflects the updated contents of the 
display information table IT including display position 
trfotlation, filed information and normal display file name and 

:rdispiay update request . - - 

controller 50, ^ rie:^;re 

rhl:" Tnrs to lie ai:: mto L mam memory 5. The 

Ti p arco tiler 5. then updates the oh^ect data in accordance 

Z tL display update data from the system controller 50 and 
Tds tLtr/upaLa oh,ect aata into the frame 

thereby to display the object on the display -^^^ <^^'- ^> ' 
a manipulated by the operator on the ^^^^ 
Thus, the system of the present invention determines a 
manipraUon to be conducted on the displayed object ^sed on 
the Luch screen information 21 which results from an operator 
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(see, Fig. 7(b)) wni ^^^^ displays the 

Pick manipulation (see rigs.. 
' A picK Banipulation is conaucted in such a way that an 
ohlect is picHea up at a position on the display surface of 
object P another position thereon. 

.i.play un.t 3 and pXaced ^^^^^^^^^^^ ^ ^^^^ manipulation. 

s. e a, ana e<h, are related and interconnected dia,ra.s 

^ fi/c\ is a table explaining, a pick 

illustrating, and Fig. 6(c) is a tao 

manipulation. „„^^iT,n to the 

. 1 4.4»n 4« carried out according ro x.ne 
A pick manipulation is carrieu « 

following steps (S1-S8) in Fig. 5: 

fSl) The system controller 50 receives 
including two sets of touch coordinates, fro„ the touch 

!yste. controller 50 ,Fi,. 3) checKs whether the 
;i,tion is a pick manipulation, as shown in Fig. 

I tahle 1. Shown in Fig. e<c, when - «la^- - 

! not a picK manipulation, the system controller 50 checks 
^ 4. fr^i- other possible manipulations. 

^°"=\::rirtreVeratio: is a pick - - 

controller 50 sends a display update request "O.-^™ 

— ; r •^r.r » 

the display information "ble IT. ^ 

^QA\ The system controller 50 receiver « ^ , 

Te system controller 50 determines whether the touch 
repor 3L includes a "continuous touch end" ---Jf" 

^ nr«i«tion is as shown in Fig. 6(b). wnen a 
:rt=t:: ::rrreported. the operation goes to step 
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(56) Otherwise, the system controller 50 updates the display 
position information "coordinates (X, Y)" of the display 
rfor^Ition tahle IT so that the oh^ect is positioned het«een the 

two fingers of the operator. 

(57) The system controller 50 sends display update request 
40 to the display controller 52, commanding that the 
social-state file be displayed according to the dxsplay 
information table IT, as updated, and returns to step (S4) . 

(58) When -continuous touch end- is reported by a touch 
report 3R, the system controller 50 sends a display update 
reLst 4Q to the display controller 52, commandxn, that the 
nt^l-display file be displayed at the position spec.f xed xn the 

''^'Z:t^^Z^or. are carried out in the same way 
as descried in the above flowchart of the pick manipulation. 
,3) scroll manipulation (see Figs. 7(a) and 7(b)). 

A scroll manipulation is conducted in such a way that an 
object, extending outside of the display surface of the dxsplay 
unit 3, is moved into and out of the , 

Fig. 7(a) is a diagram illustrating, and Fig. 7(b) is 
table explaining, a scroll manipulation. 

on determining that the operator's finger moves while 
touching the touch screen 11, based on the touch 
information 21 from the touch screen unit 1, the discriminator 
sends, to the system controller 50, a 

previously discussed including, e.g., "continuous touch start 

for the "touch type" and also, e.g., "coordinates (800, 800) 

Z touch position. AS another touch screen information 21 comes 

T: L diLiminator 51 sends a touch --^J^ J— ^ 

-continuous touch in progress- and "coordinates 

,i e the movement to the left by the finger as shown in Fig. 

;r,'and thus from X = SOO to X = 780 while at a fixed V =800). 
^el the touch screen information 21 is not sent for more than 
To mUliseconds, for example, the discriminator 51 sends a touch 
report 3R including "continuous touch end- and, e.g., 



..coordinates (700, 800," (i.e., the final X,Y coordinate as of 
the "touch end") to the system controller 50. 

""^ »,en a "continuous touch start" is reported and «>e "cb3ect 
type" is defined as "out-screen- in the display xnforBat.on table 
.fas sho«n in Fig. 7(b,, the system controller 50 "=0^"-^^; 
lipulation as a scroll and the oh^ect as a ^"^l^ ^'^jf 
beyond the display screen. Then, the system contro ler 50 
determines the speed at which the finger has moved from "^ht to 
left for example, based on a change in the x-coord.nate xn terms 
of dlta, between a touch report 3R and the following one 

Deeding on whether the finger has moved at a speed of more 
(high-speed, or less (normal-speed, than 20 dots, for example, 
the obiect display position on the display screen .s scrolled 
rnitfa ly at corresponding intervals of 100 or 500 milliseconds, 
reipiciively. Then, the interval, at which the display update 
resist 40 is sent to the display controller 52, is increased by 
aTactor of 1.5 at each touch report 3R and, when the interval 
reaches 2 seconds, the scrolling is stopped. 

practically, the display position area of the screen is so 
controlled that it starts scrolling at the appropriate -P-^; J 
above-mentioned, after the operator's finger has -^^J^/^^^f/^ 
o, 4 dots or more. That is, on recognizing that the fxnger has 
led by that distance, the system controller '^^^^^ 
"formation "display position X" of the display information table 
; so that the Object is displayed, shifted to the left by 10 
dots for example. The, it sends, to the display controller 52, 
a d Uplay updaL reguest including display ^^f^^'^^J"^^^' 

. rtianlav file name from the display 

file information and normal display 

information table IT, as updated. 

The display controller 52 reads from the hard ^^^'^ 
display file specified by the normal display file name and loads 
it in the main memory 5. The display controller 52 then 
transfers only the part of the display file, as specified by the 
transrers uiixjr r ym of' the display information 

file information "display position X' of the dispi y 




table IT, from the main memory 5 to the appropriate location of 

the frame memory 6. ^ 

in the same way, the system controller 50 sends a display 
update request 4Q to the display controller 52 every time xt 
receives a touch report 3R. 

When another "continuous touch" is reported before the 
scroll currently in progress comes to a stop, a new scroll can 
start from this point and at the first speed described above. 
(4) scroll-stop manipulation (see Figs. 7(a) and 7(b)). 

Fig. 7(a) is a diagram illustrating, and Fig. 7(b) xs a 
table explaining, a scroll manipulation. 

When a touch position given by a touch report 3R xs the same 
as or up to approximately 5 dots apart from the position of the 
scrolling currently in progress, the system controller 50 doubles 
the frequency with which display update request 4Q are sent to 
the display controller 52, in order to put an end to the 
scrolling. 

(5) push manipulation (see Figs. 8(a) to 8(c)). 

A push manipulation is conducted in such a way that an 
object is pushed on the display surface of the display unit 3. 

Fig. 8(a) is a diagram illustrating, and Fig. 8(c) xs a 
table explaining, a push manipulation. 

The system controller 50 determines the type of a 
manipulation, based on the touch report 3R and contents of the 
display information table IT shown in Fig. 8(c). When the 
manipulation is a push manipulation as shown in Fig. 8(a), the 
system controller 50 sends, to the display controller 52, a 
display update request 4Q including display position xnformatxon, 
file information and normal display file name so that the object 
is displayed close to the finger position reported by the touch 
report 3R. The above display operation is repeated until a 
"continuous touch end" is reported by a touch report 3R. 
(6) Push-while-rotate manipulation (see Figs. 8(b) and 8(c)). 

A push-while-rotate manipulation is conducted in such a way 
that an object is pushed at a position off its center (or the 
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. af>s\\ and it moves rotating 

center of gravity shown as X xn Fx,. 8(b)) and 
on the display surface of the display ^ 

^«r,<-T-oiler 50 determines the type or 

. Tatrorhareron he touch report 3R and contents of the 
Banxpulation, based ^^^^ 

display ":"'^^';i^^Z-.ot.t. manipulation as shown In Fig. 
manipulation is a P"^'' whil controller 
S(b), the system controller 50 sends 

display update requests « «^;- ^ng the angle In the 
increasing by 2 degrees, i.e., while i 

---V --"-roue" f rad^tl^^^^^^^^^ from the 

The display " 5, rotates the 

hard disk and loads the data in tn ^^j^.tcp coordinates 

Object by the specified angle and ^^/^^ ^J^,^^,,, 

,X, V) as a rotational ^^J^^^^^^, Lject rotated, from 

request 4Q, and transfers the data, witn 
the main memory 5 to the frame memory 6. 

• ,.n„„ (see rlas. 9(a) to 9(c)). 

^7^ Flip manipulation (see rig£». \ / 

I f!ip manipulation is conducted in such a way that a^ 
operator-s finger flips -.-^^a^igr peed n ;;e tluch Lreen 

Object from a .'"""""/^J.^f,,:;,:' surface of the 

11 with a corresponding result on the display 

1 ng^l^a^is a diagram Illustrating, and Fig. 9<c, is a 

' table e^cplainlng, a flip -'•^f;^;^"";^^^ ,„„ch discriminator 
When a touch '^^^^Jl^l^^tir^,,^ the type of the 

51, the system /"J^^eport 3R and the contents of 

manipulation, based on the touch rep 

the display ^"'"-^^^^""riPulalionTs shown in Fig. 9<a, , the 
manipulation is a flip """^^P^'f on the touch 
system controller 50 obtains the finger speed 
/eport 3R and also obtain ob,ect sp- i.e the^ J^^ 

intervals at which display ^ ,tem (3). 

.isplay centre e > - ^^e ^^^^ ^^^^^ ^ 

ZXZZT^r.. While updating the display position 



information left-top coordinates (X, Y) of the display 
information table IT so that the object moves in the direction 
the finger moves. The system controller 50 stops moving the 
object when the above-mentioned interval reaches 2 seconds. 
(8) Flip-under-gravity manipulation (see Figs. 9(b) and 9(c)). 

A flip-under-gravity manipulation is conducted in such a way 
that an object, which is subjected to gravity, is flipped by a 
finger on the touch screen 11 and with a corresponding result on 
the display surface of the display unit 3. 

Fig. 9(a) is a diagram illustrating, and Fig. 9(c) is a 
table explaining, a flip manipulation. 

When the finger manipulation is a flip as in the above item 
(8) and the display information table IT defines the object type 
as "gravity meaning that the object is subjected to gravity, for 
example, the object moves while receiving a combination of the 
forces of inertia and gravity, as shown in Fig. 9(b). Therefore, 
the system controller 50 sends display update requests 4Q to the 
display controller 52, while updating the display position 
information left-top coordinates (X, Y) by adding a value to the 
Y-coordinate of the display information table IT. The value is 
represented by 2 to the Nth power (N: the number of display 
update requests 4Q which are sent). In this case, too, the 
system controller 50 stops moving the object when the 
above-mentioned interval reaches 2 seconds. 
(9) Roll manipulation (see Figs. 10(a) and 10(b)). 

A roll manipulation is conducted in such a way that a 
reliable object is rolled by the operator's finger on the touch 
screen 11 and with a corresponding result on the display surface 

of the display unit 3. 

Fig. 10(a) is a diagram illustrating, and Fig. 10(b) is a 
table explaining, a roll manipulation. 

When a touch report 3R is input from the touch discriminator 
51 and the display information table IT defines the object type 
as "rollable", meaning that the object is constructed such that 
it rolls when pushed by a finger, like a globe or a cylinder and 
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as shovm in Fig. 10(a), the system controller 50 senas display 
: revests .0 to tne aisplay controller f 

the display position information left-top coordinates (X, V) 
^e display information ta.le IT so that the "-"^ 
distance 10 percent behind the distance moves on the display 
screen and in the direction the finger moves, ^-e system 
controller =0 sends display update request 4Q to «>^^-Pl^^ 
controller 52 until "continuous touch end" is reported from the 

touch discriminator 51. . ^ ^ 

,10) Distort-restore manipulation (see Figs. 11(a) and 11(b)). 

Tdistort-restore manipulation is conducted in such a way 
that an elastic object is pressed by a finger on the display 
surface of the display unit 3. 4. . 

Fig. 11(a) is a diagram illustrating, and Fig. 11(b) is a 
table explaining, a distort-restore manipulation. 

21 a touch report 3R is input from the touch discriminator 
SI and the display information table IT defines the object type 
as -elastic", meaning that the object can be 

restored according to a pressure applied thereon by a '"^ 

Is Shown in Fig. 11(a), the system controller 50 calculates an 

aLunt Of distortion of the object based on the P--"«^;'^f 

hy the touch report 3R. It stores in the display 

table IT, a special-state file name specifying a -^-^"^ 

one of the special-state files (for displaying a 

Of the object in a turn-over indication, ^"-^""^^"^ol^r 50 

amount of distortion calculated. Then, the system =<• 

sends a display update request 4Q to the display controller 52, 

commanding that the special-state file be displayed at the 

current display position. The above operation is repeated as 

necessary and when a -continuous touch end" is reported by a 

touch reU 3K, the system controller 50 sends a display update 

request 4Q (with a normal display file name 

display controller 52, commanding that a normal display file 

tLrmal indication) be displayed at the current <»-P^''>' 

1 plurality of special-state files are provided in the hard disX 
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, corresponding to the a-ount of distortion of the object which 
results from the pressure applied on the touch screen 11. 

AS is apparent from the above description, the present 
invention regards a display screen as a visual space. It defines 
conditions and physical properties of an object (e.g., werght, 
hardness, frictlonal resistance, center of gravity) in the 
display information table IT. It also receives touch screen 
information 21, indicating a finger-touched position and 
pressure, as Input from a touch screen unit 1. Based on the 
touch screen information 21 and the display information table IT, 
the present invention determines a manipulation to be conducted 
on the displayed object, e.g. scrolling, picking, P'''''^"^, 
rolling, distorting the object on the display surface of the 
display unit 3. Thus, the present invention allows a user to 
manipulate an object displayed on a display device quite easily, 
even when the user has little knowledge of computers. 
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